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This paper examines the concept of performance support systems in university teaching. In four different 
studies a performance support system with education elements was built to support students while they 
learn and work on their assignments. Performance support systems originally originated from business and 
industry but seem to be very valuable in university settings (and therefore for HRD courses) as well. The 
four different studies are evaluated and results are reported. 
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Determining the most effective and efficient conditions for supporting the performance of learners always has been 
considered as a critical issue in the research agenda of instructional design paradigms (Brown, Collins & Duguid, 
2002; Gane 1985; Mager, 1997; Merrill, 1983, Reigeluth, 1983; Spiro & Jehng, 1990; van der Meij & Carrol, 1998; 
van Merrienboer, 1997). Mager (1997) called his classical instructional design theory ‘performance oriented 
instruction’. Merrill (1983) proposed a ‘performance-content matrix, which was serving as a framework for 
designing instruction based on the component display theory. Some of the contemporary instructional design 
theories such as modeled-centered instruction (Gibson, 1998), cognitive apprenticeship approach (Brown, Collins & 
Duguid, 2002), cognitive flexibility theory (Spiro & Jehng, 1990), and minimalism (Van der Meij & Carrol, 1998) 
focused their attention on the performance support of higher-order cognitive skills. 

Although in universities it is often unclear on what analysis the curriculum is based, and even more unclear 
which instructional design paradigm is used, we believe to see a tendency towards performance based education. We 
observe many attempts to, for example, support students while they are learning and at the same moment working 
on assignments. However these attempts often do not address the complexity of the issue of performance support in 
term of performing on authentic problems, providing just-in-time, just enough, and at the point of need support, 
applying adequate performance-oriented assessment methods, and using the last advancement of the information and 
communication technology development, such as internet. An instructional design approach that brings new 
perspectives to the issue of supporting learners while performing complex tasks, is the idea of ‘performance support 
systems’. It integrates conceptually and defines operationally the notions of ‘performance’, ‘support’ and 
‘technology system’ The Internet-based performance support systems approach in university teaching follows the 
idea of electronic performance support system (EPSS), which has established a stable tradition in the domain of 
corporate management and consulting business. (Gery, 2002; Greenberg & Dickelman, 2002; Raybould, 2002). 
EPSS has been considered as a reconceptualization of both work and training environments. The idea of EPSS put 
together the worker, the learner, and the work situation in an integrated whole. (Laffey, 1997). The shortest, but the 
most conceptually rich definition of EPSS is just in time, just enough, and just at the point of need computer support 
for an effective and efficient job performance. This paper introduces and empirically validates the concept of 
‘performance support systems with so called educational elements’, that is to emphasize the main goal, namely 
teaching students. The research question is as follows; What are the effects of internet-based performance support 
system with educational elements (IPSS_EE) on the performance and attitudes of university students when given 
complex learning tasks? The research problem in general is important, also in the field of HRD, because EPSS’s are 
already used a lot in business and industry (so the context of HRD) but not so much to teach HRD courses (so HRD 
as topic). 

Performance support with Educational Elements 

Until very recently the only group in university teaching seen as a target for promoting the idea of performance 
support systems was the group of university instructors (De Croock, Paas, Schlanbusch & van Merrienboer, 2002; 
Gettman, McNelly & Muraida, 1999; McKenney, 2001; Merrill & Tompson, 1999; Nieveen & Gustafson 1999). 
Developing performance support systems for instructors follows the same motives that have driven the idea of EPSS 
in industry. There were very few attempts concerning the theory, research and practice of performance support 
systems for students. (Stoyanov, 2001). In order to provide a comprehensive definition of the concept of 
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performance support with educational element (PSS_EE) we define first operationally each of the components of 
this concept - ‘performance’, ‘support’ and ‘system’ in the context of higher technological education. 

Performance The concept of performance support with educational elements (PSS_EE) suggests practical measures 
for reducing the gap between education and its “zone of proximal development’ (Vygotsky, 1978), in term of 
defining the requirements of the future working environment after leaving the university. It implies identifying the 
‘reference situations’ for a particular education (for example HRD practitioners), meaning the domains where 
students are going to apply what they have learned. From this perspective, ‘performance’ in the conceptual 
configuration of ‘performance supports' for learning purposes can be operationalized as defining a set of authentic 
problems and tasks that cover the requirements of this working environment. A further operationalization of 
‘performance’ reflects shifting the focus from the lower levels of the learning taxonomy (Bloom, 1956) such as 
knowledge and understanding, towards its higher levels of skills' application and solving real world problems. 
Support ‘Support’ in the formation of ‘performance support systems’ for learning purposes can be operationalized 
through the following instructional design solutions: a) designing a sequence of easy-to-complex tasks; b) creating 
opportunities for practicing these tasks; c) gradually diminishing the amount of support (scaffolding); d) constant 
access to learning resources; e) variety of instructional stimuli (resources); and f) formative performance feedback 
and assessment. Ideally the problem to be solved is divided into a sequence of easy-to-complex learning tasks, 
which brings variety of experience. The instructional support for learning tasks gradually diminishes as the tasks are 
progressing to the end, the effect known as scaffolding. Practicing does not suggests that students have to learn first 
something and then apply that knowledge and skills, but rather learning while practicing a set of learning tasks, 
which lead eventually to solving real-world problems. Providing variety of instructional stimuli applies a particular 
structure of the learning resources, which consists of the following categories: a) background information with facts, 
definitions, principles and theoretical frameworks; b) examples in the format of so called ‘worked-out examples, 
modeling examples, demonstrations, and simulations; and c) procedures and techniques. Students can select at each 
moment of need one of or a combination of several instructional types of support, as the order can also be different. 
Some students may wish to start with background information, other may prefer to look first at examples, a third 
group may begin with selecting techniques and procedures. The results of performance support treatment can be 
measured through adequate summative assessment methods. Some of them, but not limited to, are constructed 
response, test lets, hands-on test, walk-through performance, ratings, portfolios, and assessment centers. Feedback 
provides a formative evaluation as informing learners about how well they have performed a particular task, what is 
the next step, and gives recommendations for remediation based on students' progress and learning preferences. 
System The term ‘system’ suggests the design and development of software applications using the recent 
developments of information and communication technologies (ICT). Performance support should be embedded into 
the interface structure, content, and the behavior of the application (Gerry, 2002). ICT is important, without this sort 
of technology we would not be able not apply completely the idea of just in time, just enough and at the point of 
need performance support. To this end we have developed an Internet-based performance support system with 
educational elements (IPSS_EE). The features of this web-based application implements the characteristics of the 
performance support construct as described above. In order to verify empirically the concept of Internet-based 
performance support systems with educational elements, we organized and conducted action research studies in four 
project sites - Plovdiv (Bulgaria), Sofia (Bulgaria), Madrid (Spain) and Grenoble (France). 

Research Design and Hypothesis 

In term of action research (McNiff, Fornax, & Whitehead, 1996; Zuber-Skerritt, 1982;) we investigate the 
effectiveness of the IPSS_EE instructional method in real educational settings (universities) and processes with 
intact groups of students. It makes the research ecologically valid. At the same time we imposed some control on 
the experimental conditions applying different types of experimental design and randomization at a certain points of 
the research process. In general there were two types of research design: a) pre-test, post-test experimental and 
control group experimental design, and b) pre-test, post-test experimental group only design. They allow the 
following types of measures: a) a comparison between the performance test results and reflections of students in an 
experimental and a control group; b) the reflections of an experimental group against a criterion including the 
characteristics of the concept of IPSS_EE; c) knowledge test and performance test; and d) an initial performance test 
without IPSS_EE and a final performance test with IPSS_EE. In addition we included in the research design a 
variation in term of subject matters, study experiences, and types of IPSS_EE. In general we test the hypothesis that 
the experimental group, working with PSS-EE will score significantly higher than the control group, which work 
under the classical conditions, on tasks performance and students’ attitudes toward the method. The following 
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section describes the specification of the experimental design and method, and presents the data in the each of the 
project sites: Plovdiv, Madrid, Sofia, and Grenoble. 

Results and Findings 

Experiment 1 Plovdiv (Bulgaria): The independent variable of the experiment is the method of instruction with two 
levels namely classical instruction and PSS_EE. Classical instruction consists of face-to-face lectures and laboratory 
exercises. In addition there is a web page with description of the task, instruction how to perform it, and reference 
information. The PSS_EE method requires students in the experimental group to use the PSS_EE prototype. There 
are two dependent variables: performance of students on tasks and their reflections about the instructional method. 
We control the possible effects of students’ experience with computers. We test the hypothesis that the experimental 
group, working with PSS-EE will score significantly higher than the control group, which work under the classical 
conditions, on tasks performance and students’ attitudes toward the method. For this purpose we apply a pre-test, 
post-test with a control and an experimental group experimental design. Forty (N = 40) first year students studying 
the second semester course ‘Information technology for physicists’ were divided equally to form two groups, which 
were then randomly assigned to the experimental and the control conditions. Each of the groups included 20 
students. The distribution pattern of their study achievements during the first semester was similar. For the purposes 
of this experiment we have developed three measuring instruments: pre-session attitude questionnaire, reflective 
questionnaire, and a performance test. The pre-session questionnaire is aimed at measuring the students’ attitudes 
towards computers in general and computer-based learning in particular. This questionnaire included 18 items. Nine 
of them were intended to check the attitudes of students toward computers and get a reliability score of .72 
(Cronbach alpha). The remaining 9 items completed the ‘learning-by-computer' scale and receives a reliability 
coefficient of .81. The post-session questionnaire reflects the characteristics of the PSS_EE as they were described 
in the previous section. It consisted of 20 items, 9 of which were indicative for ‘performance’, and 11 - for 
‘support’. The ‘performance’s items reached the reliability of . 71, while the ‘support's items went up to .76. The 
format of both questionnaires proposed a list of statements and students were asked to identify the extent to which 
they agreed with particular statements on a 5-grades Likert scale. An example is the following statement: “Learning 
resources were available all the time”. Prior to the treatment, both the experimental and the control group filled in 
the pre-assessment questionnaire. Then subjects in the experimental group worked with the PSS_EE. The control 
group followed the classical method of instruction. At the end of the study period all students did a performance test 
and filled in the reflective questionnaire. 

The data confirmed the hypothesis that the experimental group using the performance support system will score 
significantly higher then the control group, which worked under the traditional treatment conditions - F (1, 38) = 
9.875, p = .003). The confidence alpha level of .05 was used for all statistical tests. We checked the data from the 
pre-session questionnaire to be sure that the size of the effect is due to the performance support system method and 
not to some other factors, such as attitudes towards computers or learning experience with computers. The 
significance test indicated a systematic difference between the experimental group and the control group on the 
‘learning-by-computer’ scale (F (1, 38) = 7.914, p = .008). A possible explanation might be that the experimental 
group scored higher than the control group because the students allocated there have had experience with using 
computers for educational purposes while the control group had not. We applied a regression analysis to determine 
how much ‘learning-by-computer’ experience of the experimental group changes the size of the main effect 
generated by performance support system with educational elements. The data showed, after the regression, that the 
significance value due to applying IPSS_EE was still quite stable (F (2, 37) = 5.061, p = .011). The Table 1 presents 
the regression coefficients. The significant difference in the performance test’ results between the experimental and 
the control group should be attributed to using a performance support system with educational elements. The 
reflective questionnaire returns some data suggesting that this difference is due mainly to the ‘support’ factor (F (1, 
38) = 6.831, p = .013). A significant difference in relation to the ‘performance’ scale (F (1,38) = .436, p = .513) was 
not found. We also compared the reflections of the experimental group towards the characteristics of the concept of 
IPSS_EE, in term of orientation to real problems; opportunities for practicing; possibilities for getting just in time, 
just enough, and at the point of need background information examples, procedures, techniques, and just in time 
feedback; individualization; learning taxonomy’s levels (knowledge, understanding, applying knowledge and skills 
to learning exercises, and applying knowledge and skills to real world problems). The mean figures for all project 
sites are given in table 2. The experimental group in Plovdiv scores high on orientation for solving real world 
problems composed through focus on doing and problem-based organization, but it scores relatively lower on 
feeling of readiness for solving real world problems and transfer of skills The group scores high on knowing and 
understanding of content but lower on applying knowledge to learning exercises and applying knowledge to real 
world problems. The experimental group scores high on getting background information but lower on getting 



7-2 




167 



procedures and examples. It scores high on opportunities for practicing, run-time remediation, applying the method 
to other subjects matter, availability of resources, possibilities for individualization, evaluation of skills, and 
feedback, but get lower scores on structuring learning tasks. 

Table 1. Summary of regression analysis for variables predicting the significance of the performance test scores 
Coefficients a 



Variable 


B 


SE 


6 


Treatment 


-0,93 


0,36 


.014* 


Learning-by-computer 


0,19 


0,30 


.53 



Note, a Dependent variable: test scores 
* p < .05 

We also compared the reflections of the experimental group towards the characteristics of the concept of IPSS_EE, 
in term of orientation to real problems; opportunities for practicing; possibilities for getting just in time, just enough, 
and at the point of need background information examples, procedures, techniques, and just in time feedback; 
individualization; learning taxonomy's levels (knowledge, understanding, applying knowledge and skills to learning 
exercises, and applying knowledge and skills to real world problems). The mean figures for all project sites are 
given in table 2. The experimental group in Plovdiv scores high on orientation for solving real world problems 
composed through focus on doing and problem-based organization, but it scores relatively lower on feeling of 
readiness for solving real world problems and transfer of skills The group scores high on knowing and 
understanding of content but lower on applying knowledge to learning exercises and applying knowledge to real 
world problems. The experimental group scores high on getting background information but lower on getting 
procedures and examples. It scores high on opportunities for practicing, run-time remediation, applying the method 
to other subjects matter, availability of resources, possibilities for individualization, evaluation of skills, and 
feedback, but get lower scores on structuring learning tasks. 

Experiment 2 Madrid (Spain) This experiment was organized in the university for distance education in Spain 
(UNED). The students in this institution combine work and study. Because of this reason, courses dropout rate is 
relatively high. The experiment was conducted during the technical course ‘Simulation with VHDL’. Twenty-two 
students subscribed for the course, but only thirteen students continued. From those, eleven students took the 
opportunity to work with IPSS_EE and two students followed the classic way of instruction. At some early stages of 
the course, two students of the IPSS_EE group and one student of the classic study group dropped out. The 
questionnaires used were the same as in the Plovdiv case. Twenty-two students filled in ‘Leaming-by-Computer’ 
questionnaire and nine students did the reflective questionnaire. In the Madrid case we compare the results from a 
test without using IPSS_EE, and the test results after using IPSS_EE. In addition we could compare the reflections 
of the experimental group towards the criterion of IPSS_EE. According to the report of the course' instructor all 
students working with IPSS_EE got very high results on the final test - seven students achieved the highest score of 
10 (A grade) and two students got a 9 (b grade). The results from the IPSS_EE test are significantly higher then the 
preliminary test, according to the paired samples T- test data - 1 (-2,79; 9), p = .021. The experimental group scored 
high on the following items of the reflective questionnaire: availability of resources; applying the method to other 
subjects-matter; uniqueness of the method; learning task structure; just in time help; applying knowledge to learning 
exercises; and understanding learning content. The experimental group scores relatively lower on the following 
items: focus on doing; getting procedures; feeling of readiness for solving real world problems; and evaluation of 
practical skills. The means of the statements can be seen in table 2. Apart from this data, the organizers of the 
experiment collected a pool of opinions of the students, which are indicative for high positive attitudes towards the 
IPSS_EE. The students found that the IPSS_EE increased their motivation for studying, improved their 
achievements, it was pleasant to work with the system, it helped self-learning of students, and it should be used in 
teaching other subjects matter. At the same time the students reports that at the beginning they needed some time 
for getting acquainted with the system. 

Experiment 5 Sofia ( Bulgaria ) The Sofia case enhanced the variability of the research conditions. The students 
work with another IPSS_EE (which was developed as a student's MSc. Thesis). In general it implements the 
characteristics of the concept of IPSS_EE. In addition we compare the results of a knowledge test and a performance 
test. It might be assumed that students working with a PSS_EE would score differently on the performance test and 
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the knowledge test. The experiment was conducted during the course ‘Very Large Scale Integrated (VLSI) Circuits’ 
for the Master degree students in Microelectronics. Ten students began the course and eight students finished it. The 
data from paired sample t- test indicates no significant difference between the knowledge test results and the 
performance test results - t (-1,08; 7), p =. 316)*. According to the results from the reflective questionnaire the 
students from the Sofia project ranked high the features of the IPSS_EE such as run-time remediation; evaluation of 
skills and just in time feedback and help. They would like to see the system implemented across the curricula. The 
students find availability of resources very important. They think that getting background information and examples 
was easier than getting procedures. The data from the reflective questionnaire suggest that IPSS_EE supports better 
the levels of learning taxonomy such as knowing learning content, understanding learning content and applying 
knowledge and skills in learning exercises than applying knowledge and skills for solving real world problems. The 
experimental subjects scores relatively lower on the statements indicative for individualization of learning and 
orientation to real problem through the statements ‘focus on doing' and ‘feelings of solving real world problems’ . 
Table 2 presents the mean figures. 

Experiment 4 Grenoble (France) The study in Grenoble was planned to include 2 experimental and 2 control 
groups. However, because of some technical difficulties related to inconsistent Internet connection and problems 
with the system’s server, we report data only from the reflective questionnaire filled by the experimental group (N = 
48). In general the mean figures are the lowest in comparison to the other cites (See table 2). 

The students in the experimental group appreciate the most availability of resources, as procedures being the best 
presented, while examples are more difficult to find. The experimental subjects have noticed the orientation of the 
system for evaluating the skills but they cannot confirm that the focus was on doing. The students do not feel ready 
yet to solve real-world problems. All the levels of learning taxonomy such as knowing, understanding, applying 
knowledge to learning excursuses and solving real-world problems are supported by the system, according to 
students from Grenoble. The system provides just in time help but it is not sufficient for the individualization. The 
students find that IPSS_EE system is not ready yet to be implemented in other subjects matter. The details of the 
mean scores are given in table 2. 



Table 2. Mean figures of items in the reflective questionnaire 



Items 


Plovdiv® 


Sofia 1 ’ 


Madrid 0 


Grenoble' 


Focus on doing 


4,2 


3,3 


2,4 


2,7 


Availability of resources 


4,3 


4,3 


4,8 


3,8 


Run-time remediation 


4 


4 


3,8 


3,1 


Applying method to other subjects-mater 


4,3 


4,1 


4,2 


2,8 


Learning task structure 


2,2 


2,9 


4,1 


3 


Getting background information 


4,6 


4,1 


3,4 


3,2 


Individualization 


4,2 


3,8 


3,8 


2,9 


Getting examples 


3,5 


4,5 


3,4 


2,6 


Getting procedures 


3,9 


3,6 


3 


3,4 


Evaluation of skills 


3,8 


4,1 


3,2 


3,5 


Just in time help 


3,9 


4 


4,1 


3,4 


Feedback 


4,2 


3,6 


3,8 


3,1 


Feelings of solving real world problems 


2,4 


3 


3,3 


2,8 


Problem-based organization 


4,2 


3,5 


3,6 


3,3 



The highest possible score can be 6. 
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Knowing content 

Understanding content 

Applying knowledge to learning exercises 

Applying knowledge for solving real 
problems 

Transfer of skills 

Uniqueness of the method 



4,1 


4 


3,9 


3,4 


4,1 


3,9 


4 


3,4 


3,7 


4 


4 


3,6 


3,6 


2,6 


3,6 


3,2 


2 


3,6 


3,9 


3,6 


2 


3,5 


4,1 


3,1 



Note. a. N = 20; b. N = 8; c. N = 9; d. N = 48 

An item presents the meaning of a statement but not the whole expression. 

Conclusions 

The empirical studies brought some empirical evidence that performance support systems could be useful for 
university teaching. The performance support systems for educational purposes introduce not only the idea of just in 
time, just enough and at the point of need support. They operationalize this idea in practical solutions. One of the 
most promising among them is structuring the resources under the categories such as background information 
(definitions, mental models, theoretical frameworks), examples (work-out example, simulations, demonstrations), 
and procedures (guidelines, techniques, software). The focus here is on the individualization issues, which are also 
very typical for HRD courses. Structuring the resources in this way provides good opportunities for so called 
embedded adaptation (Stoyanov, 2001; Stoyanov & Kirschner, 2004). The performance support system 
accommodates implicitly in its interface the learning styles (Mumford & Honey, 1992; Kolb, 1994) and the 
knowledge level of students. The functionality and the interface afford students to select what they need and when 
they needed it. For example, students having a reflective learning style could start with particular examples. Students 
with a pragmatist learning style are expected to begin with procedures. People having a theorist learning style would 
look first at the background information. Students have also the opportunity to select the level of difficulty in the 
tasks sequence according to their prior knowledge. The way of structuring learning tasks and learning resources in a 
performance support system with educational element assumes two types of situations: internal and external learning 
locus of control (Stoyanov, 2001). In an internal learning locus of control situation students are given opportunity to 
select the appropriate task and the type of supportive resources. They know what type of information they need and 
they would select the types of resource that, from one side, match to their learning preferences, and from another, 
are complementary to their learning preferences. A different type of situation is based on the idea of external 
learning locus of control, according to which students need support in selection of task and resources. They are 
advised to pick up a type of resource that fits to their learning preferences but also to look at other type of resources 
in order to make their learning preferences more versatile. This way the concept of performance support systems 
with educational elements is related to another dimension of adaptation. The idea contributes not only to the 
preferential type of adaptation but also to the development adaptation. In addition, the feedback after completion of 
a task adjusts the initial recommendations to students or corrects the choices made by them. The reflective 
questionnaire brought back a pattern of responses that favors the idea of performance support system for university 
teaching. It was a personally appealing concept for the students. In general, they showed high positive attitudes 
toward the idea and were enthusiastic about its implementation across the higher education engineering curricula. 
At the same time the data identified some issues that need a further consideration. The tested performance support 
system did not promote at the required level the idea of ‘performance’. It was operationalized through proposing 
authentic problems, building opportunities for practicing on these types of problems and applying adequate methods 
for performance assessment. According to the reflective questionnaire, students did not have feeling of dealing with 
real problems. They thought the tasks to solve were no more than a learning exercise. From one side, it is important 
to avoid confronting first year students to real world problems, or at least not at the beginning. It may increase the 
cognitive load, which has proved counterproductive in learning situations (Merrienboer, 1997). From another side, 
the learning tasks can look like real problems when providing a context in an appropriate format. Learning cases 
can be good examples. In general, more effort is needed to construct different formats of problems for learning 
purposes. Related to this issue is the fact that in most of the cases the students identified knowledge and 
understanding as the levels of learning taxonomy they reached, but not the level of applying knowledge and skills 
for solving real world problems. In general, the reason that IPSS_EE did not produce a significant effect on the scale 
of ‘performance’ is due not to the idea behind ‘performance’, but to the realization of the idea and its 
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implementation in the software application. Although the results of the experiment raised the optimism for 
introducing the idea of internet-based performance support systems in university teaching, the outcomes should be 
carefully generalized to other settings. The action research type of the studies contributed to the ecological validity 
of the research, but produced some negative effects in term of controlling the experimental conditions. We designed 
an experiment, developed instruments and gave an instruction to university instructors how to conduct the 
experiment, but we did not know actually what happened during the experimental session. The measuring 
instruments were far from perfect. The questionnaires were translated in Bulgarian, French and Spanish but no back 
translation was made. Nevertheless all shortcomings we think that this large research project in three different 
countries contributes a lot to a better understanding of performance support systems in university teaching. For the 
field of HRD it would be nice to start an experiment with an IPSS_EE in HRD courses. We believe that it would be 
a great help to both students and university instructors. 
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